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Introduction 


Background 

During  the  combat  operations  phase  of  Operation  Iraqi  Freedom  (OIF),  Anny  Patriot  units  were 
involved  in  two  fratricide  incidents.  In  the  first,  a  British  Tornado  was  misclassified  as  an  anti¬ 
radiation  missile  (ARM)  and  subsequently  engaged  and  destroyed.  The  second  fratricide 
incident  involved  a  Navy  F/A-18  that  was  misclassified  as  a  tactical  ballistic  missile  (TBM)  and 
also  engaged  and  destroyed.  Three  flight  crew  members  lost  their  lives  in  these  incidents.  OIF 
involved  a  total  of  1 1  Patriot  engagements  by  U.S.  units.  Of  these  11,  nine  resulted  in  successful 
TBM  engagements;  the  other  two  were  fratricides. 

Patriot  is  the  Army’s  first-line  air  and  missile  defense  (AMD)  system.  The  system  has  been  in 
the  active  force  since  the  early  1980s.  Initially,  Patriot  was  intended  as  a  defense  against 
conventional  air-breathing  threats  (ABTs).  However,  since  Operation  Desert  Storm  (ODS)  in  the 
early  1990s,  the  system  has  been  used  primarily  against  TBMs.  Future  usage  scenarios  envision 
the  system  being  used  against  a  spectrum  of  air  threats  including  TBMs,  conventional  ABTs, 
cruise  missiles,  and  various  categories  of  unmanned  aerial  vehicles  (UAVs).  The  range  of 
potential  air  threats  in  the  contemporary  operating  environment  has  significantly  increased  the 
complexity  of  the  battle  command  problem  for  Patriot  and  other  AMD  systems. 

Since  Patriot  is  an  existing  system  and  has  been  in  the  Army’s  inventory  since  the  early  1980s, 
what  do  lessons  from  Patriot  tell  us  about  other  systems,  particularly  those  at  the  concept  stage  or 
under  development?  As  Patriot  has  evolved  over  the  past  two  decades,  the  system  has  acquired 
features  and  characteristics  that  are  more  typical  of  systems  the  Army  will  employ  in  the  future 
than  those  in  the  current  inventory.  Terms  that  are  now  used  to  describe  Patriot  include  (1)  joint, 
(2)  network-centric,  (3)  complex,  and  (4)  knowledge-intensive.  First,  command  and  control  (C2) 
for  the  Patriot  system  is  joint — involving  both  the  Army  and  Air  Force,  and  sometimes  the  Navy. 
Second,  effective  employment  of  system  assets  is  dependent  on  a  robust  network.  Third,  the 
system  as  broadly  defined  is  complex  in  that  it  consists  of  a  large  number  of  interacting 
components.  And  fourth,  Patriot  is  knowledge-intensive  in  terms  of  the  amount  of  information 
required  to  characterize  and  comprehend  the  system.  It  can  be  argued  that  Patriot  is  a  relevant 
data  point  regarding  human  perfonnance  issues  the  Army  is  likely  to  face  with  emerging  high 
technology  and  network-centric  systems.  Moreover,  this  glimpse  into  the  future  is  tangible  and 
real  and  not  abstract  or  hypothetical.  The  lessons  discussed  in  this  paper  are  from  the  crucible  of 
combat  operations  and  not  based  solely  on  the  results  of  operational  tests  or  simulated  exercises. 

The  central  focus  of  the  paper  is,  however,  human  system  integration  (HSI)  or  MANPRINT 
(Manpower  and  Personnel  Integration)  observations  and  lessons  from  the  Army’s  25-year 
developmental  experience  with  Patriot.  [Note:  MANPRINT  is  the  Anny’s  HSI  initiative.  The 
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program  was  started  several  years  after  Patriot  was  initially  fielded.]  This  retrospective  case 
study  assessment  of  HSI  practices,  successes,  failures,  and  lessons  was  made  possible  because  of 
a  unique  research  opportunity  that  resulted  from  the  Patriot  fratricides  during  OIF :  The  Patriot 
Vigilance  project.  The  next  section  provides  an  overview  of  the  Patriot  Vigilance  project  and  its 
results.  Prior  to  proceeding,  it  should  be  noted  that  the  discussion  to  follow  is  presented  in  the 
spirit  of  action  research,  defined  as  self-directed  research  that  any  community  of  practice  can  do 
to  improve  its  methods  with  the  aim  of  improving  its  strategies,  methods,  and  knowledge  of  the 
environment  in  which  it  practices  (Schein,  2004).  The  intent  of  the  report  is  institutional 
learning  and  HSI  practice  improvement  rather  than  simple  after-the-fact  criticism. 

The  Patriot  Vigilance  Project 

Personnel  from  the  Anny  Research  Laboratory’s  Human  Research  and  Engineering  Directorate 
(ARL  HRED)  began  looking  into  Patriot  and  AMD  perfonnance  and  training  issues  at  the 
invitation  of  the  then  Ft.  Bliss  Commander,  Major  General  (MG)  Michael  A.  Vane.  MG  Vane 
was  interested  in  operator  vigilance  and  situational  awareness  (SA)  as  they  relate  to  the 
performance  of  automated  AMD  battle  command  systems.  [Note:  The  generally  accepted 
definition  of  SA  is  from  Endsley,  Bolte,  and  Jones  (2003)  who  define  it  as  the  perception  of 
elements  in  the  environment,  the  comprehension  of  their  meaning,  and  the  projection  of  their 
status  in  the  near  future.]  MG  Vane  was  particularly  concerned  by  what  he  termed  a  “lack  of 
vigilance”  on  the  part  of  Patriot  operators  along  with  an  apparent  “lack  of  cognizance”  of  what 
was  being  presented  to  them  on  situation  displays  and  a  resulting  “absolute  trust  in  automation.” 
His  request  for  human  factors  support  was  prompted  by  the  unacceptable  rate  of  fratricidal 
engagements  by  Patriot  units  during  OIF — two  out  of  a  total  of  1 1  engagements,  or  18%.  MG 
Vane’s  reference  to  lack  of  vigilance  by  Patriot  operators  led  to  the  effort  being  called  the  Patriot 
Vigilance  project. 

Following  general  approaches  to  human  error  investigations  and  case  study  research  outlined  in 
Dekker  (2002)  and  Yin  (2003),  respectively,  the  project  staff  spent  most  of  the  summer  and  fall 
of  2004  performing  a  human-perfonnance-oriented  critical  incident  assessment  of  the  OIF 
fratricides.  This  involved  activities  such  as  reading  documents  from  the  fratricide  boards  of 
inquiry  (BOIs),  interviewing  knowledgeable  personnel  in  the  Ft.  Bliss  area,  and  observing  Patriot 
training  and  operations.  An  initial  assessment  briefing  was  delivered  to  MG  Vane  in  October 
2004.  The  project  staff  also  prepared  a  supporting  technical  report  describing  the  human 
performance  problems  associated  with  automation  and  supervisory  control  (Hawley,  Mares,  & 
Giammanco,  2005). 

The  logic  model  (see  Yin,  2003)  resulting  from  HRED’s  critical  incident  assessment  of  the  OIF 
fratricides  is  presented  in  figure  1.  In  developing  this  logic  model,  the  project  staff  was 
responding  to  MG  Vane’s  request  to  explain  “how  we  got  to  the  OIF  incidents.”  He  was  not 
interested  in  a  further  dissection  of  the  specifics  of  the  incidents,  since  those  details  had  been  the 
focus  of  the  various  BOIs  convened  to  examine  the  fratricides.  Rather,  MG  Vane  was  interested 
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in  understanding  how  the  branch  got  into  a  situation  in  which  those  incidents  were  almost 
inevitable.  His  observations  concerning  Patriot  operator  perfonnance  cited  above  speak  directly 
to  the  General’s  view  of  the  situation.  The  logic  model  was  constructed  with  this  intent  in  mind. 
It  is  explanatory  in  a  broad,  conceptual  manner  rather  than  in  a  narrow,  technical  sense  (e.g.,  a 
step-by-step  dissection  of  operator  actions).  HRED’s  intent  also  was  to  point  the  way  to 
actionable  solutions  rather  than  to  lay  further  blame. 


The  first  block  in  the  causal  network  leading  to  the  OIF  fratricides  is  termed  “undisciplined 
automation,”  defined  as  the  automation  of  functions  by  designers  and  subsequent  implementation 
by  users  without  due  regard  for  the  consequences  for  human  performance  (Parasuraman  &  Riley, 
1997).  Undisciplined  automation  tends  to  define  the  operators’  roles  as  by-products  of  the 
automation.  Operators  are  expected  to  “take  care  of’  whatever  the  system  cannot  handle. 
However,  in  the  case  of  Patriot,  little  explicit  attention  was  paid  during  design  and  subsequent 
testing  to  determining  (1)  what  these  residual  functions  were,  (2)  whether  operators  reasonably 
could  be  expected  to  perfonn  them,  (3)  how  operators  should  be  trained,  or  (4)  the  impact  on  the 
overall  system’s  (hardware  plus  operators)  decision-making  reliability. 


Little  explicit  regard  for 
the  human  performance 
consequences  of  design 
\  decisions 


Reinforced  by 
training  practices: 
Rote  drills  vs.  high-level 
judgment 


Contributed  to  by 
lack  of  experience  on 
the  part  of  personnel 
in  key  crew  positions,/ 


Undisciplined 

Automation 


Automation  Misuse: 
Automation  bias  by[ 
operators 


Vigilance  Problems 
and  Inadequate  SA— 
lack  of  comprehension 
of  tactical  situation 


U' 


OIF  Fratricide 
Incidents 


...fascination  witm 
and  “blind  faith” 
in  technology 
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quickly,  engage 
early,  and  trust  , 
the  system  w/o 
question” 


Figure  1.  Patriot  vigilance  logic  model. 

The  downstream  impact  of  undisciplined  automation  was  exacerbated  by  two  additional  factors: 
(1)  unacknowledged  system  fallibilities,  and  (2)  a  “fascination  with  and  blind  faith  in 
technology.”  [Note:  Several  tenns  presented  in  quotes  without  reference  citations  are  taken  from 
the  classified  BOI  reports.]  A  series  of  Patriot  operational  tests  indicated  that  the  system’s 
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automated  engagement  logic  was  subject  to  air  target  (i.e.,  track)  misclassification  problems — 
system  fallibilities.  However,  these  sources  of  automation  unreliability  were  not  fully  addressed 
during  system  software  upgrades,  nor  did  infonnation  about  them  find  its  way  into  operator 
training;  battle  command  practices;  tactics,  techniques,  and  procedures  (TTPs);  or  Tactical 
Standing  Operating  Procedures  (TSOPs).  System  developers  continued  to  pursue  technology¬ 
centric  solutions  to  automation  reliability  problems  (e.g.,  increased  use  of  artificial  intelligence, 
non-cooperative  target  recognition,  etc.).  But  the  basic  problem  remained:  The  total  system 
(hardware  plus  crew)  was  not  sufficiently  reliable  in  critical  functional  areas,  most  notably  track 
classification  and  identification.  Users  were  not  informed  regarding  these  problems,  or  if  they 
were  infonned,  little  effective  responsive  action  was  taken. 

In  the  aftermath  of  the  first  Gulf  War  (ODS),  the  AMD  user  community  (combat  developer, 
training  developer,  operational  units,  etc.)  acquiesced  to  the  developmental  community’s 
apparent  lack  of  urgency  regarding  problems  with  Patriot’s  track  classification  accuracy. 
Emboldened  by  Patriot’s  seeming  success  in  engaging  the  Iraqi  SCUD  threat  during  ODS, 
Patriot’s  organizational  culture  emphasized  “Reacting  quickly,  engaging  early,  and  trusting  the 
system  without  question.”  Users  were  allowed  to  persist  in  their  belief  that  the  system  would  not 
confuse  air-and  non-air-breathing  threats  such  as  ARMs  and  TBMs. 

The  cultural  norm  of  unwarranted  trust  in  automation  was  exacerbated  by  the  AMD  branch’s 
traditional  training  practices,  which  were  criticized  in  BOI  reports  as  emphasizing  “rote  drills 
versus  the  exercise  of  high-level  judgment.”  The  Patriot  user  community  continued  to  approach 
training  for  air  battle  operations  in  much  the  same  manner  as  march  order  and  emplacement  or 
system  set-up.  The  emphasis  was  on  mastering  routines  rather  than  active  thinking  and  adaptive 
problem  solving.  Klein  and  Pierce  (2001)  refer  to  the  result  of  this  practice  as 
“experiosclerosis.”  Crews  believe  they  are  competent  and  “combat  ready”  because  they  are  good 
at  the  routines,  but  the  routines  can  prove  to  be  a  straitjacket  during  combat.  Traditional 
individual  and  unit  evaluation  practices  reinforced  this  mistaken  belief  on  the  part  of  crews  and 
commanders  at  all  levels  by  focusing  only  on  satisfactory  performance  of  routine  drills.  The 
Army  BOI  investigating  the  OIF  fratricides  stated  bluntly  that  “the  system  (Patriot)  is  too  lethal 
to  be  placed  in  the  hands  of  crews  trained  to  such  a  limited  standard.” 

A  second  detrimental  factor  was  the  branch’s  traditional  personnel  assigmnent  practices  which 
tended  to  place  inexperienced  personnel  in  key  crew  positions  in  the  C2  chain:  the  battery-level 
Engagement  Control  Station  (ECS)  and  battalion-level  Information  and  Coordination  Central 
(ICC).  Before  the  first  round  was  fired  during  OIF,  the  stage  was  thus  set  for  what  Parasuraman 
and  Riley  (1997)  refer  to  as  “automation  misuse,”  specifically  automation  bias  on  the  part  of 
Patriot  operators.  Automation  bias  is  defined  as  unwarranted  over-reliance  on  automation,  and 
has  been  demonstrated  to  result  in  failures  of  monitoring  (vigilance  problems)  and  accompanying 
decision  biases  (an  absolute  and  unthinking  trust  in  automation — let’s  do  what  the  machine 
recommends).  Recall  that  these  are  the  very  concerns  expressed  by  MG  Vane  in  his  kick-off 
discussion  with  the  Patriot  Vigilance  staff. 
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One  must  be  careful,  however,  not  to  lay  too  much  blame  for  these  shortcomings  at  the  feet  of 
the  Patriot  operators  or  the  supporting  battle  staff.  As  suggested  in  figure  1 ,  the  roots  of  these 
human  shortcomings  can  be  traced  back  to  systemic  problems  resulting  from  decisions  made 
years  earlier  by  concept  developers,  software  engineers,  procedures  developers  (government  and 
vendor),  trainers,  and  commanders.  In  one  sense,  the  OIF  Patriot  operators  did  what  they  had 
been  trained  to  do  and  what  Patriot’s  culture  emphasized  and  reinforced.  It  should  also  be  noted 
that  system  developers  and  users  edged  into  the  OIF  situation  by  degrees.  As  Patriot  evolved 
over  its  2  5 -year  life,  its  dominant  control  mode  changed,  by  degrees,  from  traditional  manual 
control  to  supervisory  control  as  increasing  levels  of  automation  and  other  technical  features 
were  added  to  cope  with  increasingly  sophisticated  threats  and  an  increasingly  complex 
operating  environment. 

Hardware-wise,  Patriot  evolved  into  a  very  lethal  system.  It  can  be  argued,  however,  that  the 
system  was  not  properly  managed  during  OIF.  Driven  by  technology  and  mission  expansion,  the 
Patriot  crew’s  role  changed  from  traditional  operators  to  supervisory  controllers  whose  primary 
role  is  supervision  of  subordinate  automatic  control  systems.  But  this  role  change  was  not 
reflected  in  the  AMD  culture,  design  and  evaluation  practices,  battle  management  concepts, 
operational  procedures,  training  practices,  or  personnel  usage  patterns.  Moreover,  system 
management  issues  (doctrine,  battle  command  concepts,  TTPs,  TSOPs,  etc.)  and  crewmembers’ 
ability  to  execute  them  were  not  addressed  with  the  same  rigor  during  development  and 
evaluation  as  hardware  and  software  capabilities.  As  the  lessons  of  OIF  suggest,  these  aspects  of 
the  total  “system”  are  as  important  to  operational  effectiveness  as  hardware  and  software 
capabilities. 

HRED’s  briefing  to  MG  Vane  in  October  2004  described  the  human  performance  circumstances 
that  contributed  to  the  fratricides  and  recommended  two  primary  actionable  items  to  address  the 
problems  thus  identified: 

1 .  Re-examine  automation  concepts,  operator  roles,  and  C2  relationships  in  AMD  battle 
command  systems  to  emphasize  effective  human  supervisory  control  (HSC);  and 

2.  Develop  more  effective  missile  crews  and  C2  teams,  or  in  the  words  of  the  Anny  BOI 
report  “re-look  the  level  of  expertise  required  to  operate  such  a  lethal  system  on  the  modern 
battlefield.” 

In  present  usage,  the  tenn  effective  HSC  refers  to  a  situation  in  which  soldiers  and  not  the 
automated  system  are  the  ultimate  decision  makers  in  AMD  firing  decisions.  Uncritical 
acquiescence  to  the  automated  system’s  recommendations  is  not  effective  HSC. 

A  month  following  HRED’s  report  to  MG  Vane,  the  Defense  Science  Board  (DSB)  (DSB,  2004) 
reinforced  HRED’s  conclusions  with  the  following  recommendations.  Although  the  full  DSB 
report  on  Patriot  system  performance  is  classified,  these  extracts  are  not. 
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“The  Patriot  system  should  migrate  to  more  of  a  ‘man-in-the-loop’  philosophy 
versus  a  fully  automated  philosophy — providing  operator  awareness  and  control 
of  engagement  processes.” 


“Patriot  training  and  simulations  should  be  upgraded  to  support  this  man-in-the- 
loop  protocol  including  the  ability  to  train  on  confusing  and  complex  scenarios 
that  contain  unbriefed  surprises.” 

A  summary  of  the  DSB  report  on  Patriot  system  performance  is  available  for  download  on  the 
DSB’s  web  site. 

Follow-On  Work,  Implementation,  and  Current  Status 

After  reviewing  initial  project  results,  the  Army  Training  and  Doctrine  Command  (TRADOC) 
Capability  Manager  for  Lower  Tier  AMD  systems  (TCM-LT),  requested  that  the  Patriot 
Vigilance  project  continue  into  a  second  phase.  The  TCM  specifically  requested  that  HRED’s 
project  staff  expand  on  the  material  presented  in  Hawley,  Mares,  and  Giammanco  (2005)  and 
prepare  two,  more-detailed  reports,  one  concerned  with  design  for  effective  human  supervisory 
control  and  a  second  addressing  training  for  the  emerging  class  of  automated  AMD  battle 
command  systems.  In  the  TCM’s  words,  the  intent  of  these  reports  was  to  inform  the  AMD 
community  on  “what  right  looks  like”  in  each  of  these  topic  areas.  The  results  of  the  second 
phase  of  the  effort  were  the  technical  reports  Developing  Effective  Human  Supervisory  Control 
for  Air  and  Missile  Defense  Systems  (Hawley  &  Mares,  2006)  and  Training  for  Effective  Human 
Supervisory  Control  of  Air  and  Missile  Defense  Systems  (Hawley,  Mares,  &  Giammanco,  2006). 
Both  reports  contain  a  summary  and  discussion  of  the  technical  state  of  the  art  in  each  of  the 
topic  areas.  In  addition,  supporting  informational  briefings  were  developed  for  use  across  the 
AMD  community.  The  project  staff  also  worked  with  various  elements  in  the  AMD  system 
development  (combat  and  materiel  developer),  training,  and  user  communities  on  operationally 
defining  and  implementing  Patriot  Vigilance  recommendations.  Phase  two  formed  the 
theoretical  basis  for  what  later  were  to  be  turned  into  actual  design  and  training  modifications. 

In  the  late  summer  of  2005  after  MG  Vane  had  left  Ft.  Bliss  for  another  assignment,  the  project 
staff  briefed  his  replacement,  MG  (then  Brigadier  General)  Robert  P.  Lennox,  on  the  status  and 
results  of  the  Patriot  Vigilance  project.  Based  on  this  presentation  and  subsequent  urging  from 
the  TCM-LT,  MG  Lennox  formally  requested  that  the  project  be  continued  for  at  least  another 
year  so  that  the  technical  staff  could  continue  to  work  with  the  AMD  community  on 
implementing  selected  results.  HRED’s  project  staff  also  would  participate  as  the  MANPRINT 
evaluator  during  an  operational  test  of  the  Post-Deployment  Build  6  (PDB-6)  software  suite  for 
the  Patriot  system.  PDB-6  was  developed  to  address  many  of  the  Patriot  system’s  operational 
deficiencies  that  had  surfaced  during  OIF  and  were  generally  considered  to  have  contributed  to 
the  unacceptable  fratricide  rate. 
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It  also  turned  out  that  in  addition  to  wide-ranging  software  fixes,  the  PDB-6  operational  test  was 
expanded  to  address  a  number  of  changes  consistent  with  HRED’s  first  actionable  item 
concerning  a  re-examination  of  automation  concepts,  operator  roles,  and  C2  relationships  in 
AMD  battle  command  systems  to  emphasize  effective  HSC.  The  centerpiece  of  these  changes 
was  the  integration  of  a  Fire  Coordination  Center  (FCC)  into  the  Patriot  battalion  command  post. 
The  FCC  represents  an  enhanced  C2  entity  similar  in  concept  to  the  combat  information  center 
on  Navy  Aegis  cruisers.  If  the  FCC  concept  proved  successful,  it  rather  than  the  traditional  ECS- 
ICC  combination  would  become  the  “trigger-puller”  for  Patriot  units.  With  the  introduction  of 
the  FCC,  the  branch  was  implicitly  recognizing  that  “two  people  inside  a  van  (the  ECS) 
conducting  engagement  operations  is  no  longer  viable.”  The  FCC  potentially  represents  a 
significant  step  forward  in  addressing  the  SA  problem  that  contributed  to  the  C2  failures  of  OIF. 
First,  however,  it  would  have  to  be  demonstrated  (1)  that  the  FCC  provided  the  incremental  SA 
essential  for  more  accurate  engagement  decision  making  and  (2)  that  the  new  C2  configuration 
could  do  so  in  a  timely  manner.  Engagement  decision  time  lines  for  Patriot  against  TBMs  are 
very  short — less  than  10  seconds  in  the  case  of  the  fratricide  involving  the  British  Tornado 
during  OIF.  Decision  cycle  time  is  a  significant  issue  in  AMD  battle  command. 

From  the  fall  of  2005  through  the  summer  of  2006  during  the  New  Equipment  Training  (NET) 
and  unit  train-up  period  for  the  PDB-6  test,  the  HRED  project  staffs  observations  regarding  the 
progress  of  training  for  the  test  unit  sounded  an  alarm.  PDB-6  training  was  not  progressing 
according  to  plan.  Training  events  were  being  completed,  but  individual  and  crew  performance 
objectives  were  not  being  met.  In  addition,  many  of  the  training  issues  identified  and  discussed 
in  Hawley,  Mares,  and  Giammanco  (2006)  were  re-surfacing  and  were  not  being  addressed 
adequately  by  the  NET  process  or  follow-on  collective  training  by  the  test  unit.  These  included 
but  were  not  limited  to  (1)  an  emphasis  on  training  events  to  the  exclusion  of  test  player 
performance  capabilities,  (2)  lack  of  focus  on  the  unit’s  core  test  mission — air  battle  operations, 
(3)  inadequate  standards,  (4)  inappropriate  training  methods,  and  (5)  inadequate  perfonnance 
feedback — the  after-action  review  (AAR)  process. 

The  project  staff  viewed  these  deficiencies  as  a  serious  problem  because  inadequate  test  player 
training  would  compromise  the  validity  of  test  results  and  undermine  the  basis  for  evaluating  the 
value  added  of  PDB-6  software  changes,  the  FCC  concept,  and  other  C2  modifications  (see 
Hawley,  2007a).  Even  more  serious  was  the  fact  that  fratricides  and  fratricide-inducing 
conditions  (e.g.,  dropped  or  improperly  correlated  tracks,  loss  of  network  connectivity,  etc.) 
similar  to  those  that  occurred  during  OIF  were  still  all-too-frequent  during  the  test  itself.  Many 
of  the  human  perfonnance  problems  that  had  shown  up  during  OIF  were  apparent  again,  with 
similar  results. 

The  Patriot  and  AMD  C2  situation  is  not  all  gloom  and  doom.  As  a  result  of  the  Patriot 
Vigilance  project  and  HRED’s  participation  in  the  PDB-6  FUT,  beneficial  changes  are  in  the 
offing.  These  include  (1)  new  concepts  for  effective  HSC  for  Patriot  and  follow-on  AMD 
systems  (e.g.,  the  FCC  and  a  follow-on  AMD  integrated  battle  command  system);  (2) 
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performance-oriented  concepts  for  NET  and  unit  train-up  prior  to  testing  for  Patriot 
enhancements  (e.g.,  PDB-6.5  and  -7.0);  (3)  modified  concept  evaluation  and  test  and  evaluation 
practices;  and  (4)  a  revision  of  institutional  and  unit  training  practices  to  emphasize  deliberate 
practice,  active  thinking,  and  sensemaking  over  traditional  crew  drills.  An  overview  of  these 
developments  is  provided  in  Hawley  (2007b). 


MANPRINT  Observations  and  Broader  Lessons 


The  idea  for  a  MANPRINT  observations  and  lessons  assessment  using  Patriot  as  an  exemplar 
grew  out  of  a  conversation  with  the  TCM-LT  prior  to  the  conclusion  of  the  PDB-6  LUT.  After 
discussing  emerging  results  from  the  test  and  recommendations  from  the  Patriot  Vigilance 
project,  the  TCM  said  something  to  the  effect  that  “We’ve  been  doing  MANPRINT  work  on 
Patriot  for  more  than  20  years.  How  did  we  miss  all  of  the  things  you  wrote  about  and  that  we’re 
now  seeing?  Certainly  all  of  these  things  are  not  new.”  These  questions  prompted  the  project 
staff  to  ask  the  obvious  follow-on  questions  “How  did  we  miss  these  high-driver  issues?  Or,  did 
we  note  these  things  and  nothing  came  of  them?”  These  latter  questions  led  to  a  review  of  the 
available  Patriot-related  MANPRINT  assessments  and  test  reports  going  back  to  the  start  of  the 
formal  MANPRINT  initiative  in  the  mid-1980s.  Prior  to  MANPRINT’s  formalization  in  1985, 
human  factors  concepts  were  different.  HSI  as  we  understand  the  tenn  today  was  not  explicitly  a 
goal  in  system  development.  Rather,  the  concern  in  those  days  was  a  rather  narrow  view  of  what 
is  now  termed  human  factors  engineering  (HFE):  Properly  engineering  the  interface  between 
users  and  the  hardware  system,  or  fitting  the  person  to  the  system. 

The  next  section  presents  the  project  team’s  observations  concerning  the  Patriot  MANPRINT 
program  that  evolved  after  1985.  Readers  should  note  that  the  operant  word  in  the  previous 
sentence  is  “evolved.”  MANPRINT  concepts  as  applied  to  Patriot  evolved  from  a  nearly  strict 
preoccupation  with  HFE  to  include  a  broader  concern  for  other  nominal  MANPRINT  domains: 
Manpower,  Personnel,  Training,  System  Safety,  Health  Hazards,  and  Soldier  Survivability.  The 
project  staff  has  tried  to  avoid  hindsight  bias  and  judging  earlier  MANPRINT  activities  by  the 
conceptual  standards  of  today.  Again,  the  intent  is  practice  improvement  rather  than  mere 
criticism  of  what  went  on  with  Patriot. 

Observations  on  Patriot  MANPRINT 

A  review  of  MANPRINT  documents  and  test  reports  going  back  to  the  mid-  to  late- 1980s  and 
into  the  1990s  indicated  that  the  term  evolved  might  actually  be  a  misnomer  when  describing  the 
Patriot  MANPRINT  program.  Early  assessments  focused  on  what  might  be  tenned  “1472 
issues”  (control  and  display  features,  symbology  concerns,  etc.)  and  user-jury  type  evaluations. 
The  term  1472  issues  refer  to  the  concerns  addressed  in  MIL-STD-1472,  Human  Engineering. 
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User-jury  type  assessments  refer  to  the  extensive  use  of  questionnaires  and  interviews  during 
system  evaluation.  User-jury  data  typically  are  collected  from  a  panel  of  recognized  experts. 

There  is  nothing  inherently  wrong  with  the  assessment  methods  alluded  to  in  the  previous 
paragraph.  They  are  a  part  but  not  all  of  most  contemporary  MANPRINT  assessments. 

Problems  began  to  arise,  however,  when  our  MANPRINT  methods  did  not  evolve  as  the  system 
evolved.  Over  the  course  of  its  25-year  life,  Patriot  evolved  from  a  system  that  was  quite 
literally  just  one-step  more  advanced  than  its  predecessor,  the  Hawk  missile  system,  to  the 
complex  system  that  exists  today.  As  more  and  more  technology  and  software  capability  was 
added  to  the  Patriot  system,  the  operators’  roles  changed  from  traditional  manual  control  to 
supervisory  control,  bringing  with  it  everything  that  goes  with  that  control  mode.  Our 
MANPRINT  methods  did  not  advance  in  concert  with  system  enhancements.  We  continued  to 
look  at  the  system  as  if  it  were  no  different  than  a  traditional  manual  system.  The  MANPRINT 
assessments  performed  for  later  system  upgrades  were  not  substantially  different  from  those 
performed  early  in  the  system’s  development.  Moreover,  many  of  the  human  performance 
problems  that  became  apparent  during  OIF  and  later  during  the  PDB-6  LUT  were  starting  to 
show  up  in  earlier  test  results.  But  the  MANPRINT  community  did  not  raise  a  red  flag,  and 
potential  show-stoppers  either  were  ignored  or  allowed  to  be  downplayed.  The  most  common 
form  of  downplaying  was  to  dismiss  human  performance  problems  as  isolated  training  issues 
that  could  be  fixed  later  during  institutional  or  unit  training.  It  should  be  noted  that  most  of  the 
human  performance  problems  thus  identified  and  categorized  as  training  problems  were  never 
actually  addressed. 

The  bottom  line  with  respect  to  the  20-year  history  of  Patriot’s  MANPRINT  program  is  that  we 
as  a  community  missed  the  elephants  in  the  middle  of  the  room.  These  were  (1)  an  eroding 
potential  for  effective  human  supervisory  control,  and  (2)  the  need  for  increased  levels  of 
operator  expertise — not  simply  additional  training  of  the  traditional  kind,  but  training  with  a 
focus  on  developing  operator  expertise.  It  should  be  noted  that  the  DSB’s  post-OIF  review  of 
Patriot  system  performance  focused  quickly  on  these  issues,  as  did  the  Patriot  Vigilance 
assessment  requested  by  MG  Vane.  Furthermore,  similar  problems  were  apparent  in  related 
domains  going  back  to  near  the  time  the  Patriot  system  was  initially  fielded.  These  domains 
include  nuclear  power  operations  and  the  1979  Three  Mile  Island  incident;  Navy  anti-air 
operations  and  the  1988  USS  Vincennes-Iranian  airbus  incident;  and  the  FAA’s  $3.5  billion 
Advanced  Automation  System  (AAS)  “meltdown”  in  the  1990s.  [See  Bar- Yam  (1997)  for  a 
discussion  of  the  events  leading  to  the  “safety  veto”  of  the  AAS.]  All  of  these  incidents  were 
reported  and  discussed  widely  in  the  human  factors  literature,  yet  no  mention  of  them  is  found  in 
Patriot  MANPRINT  documents — despite  the  fact  that  Patriot’s  operational  concept  has  many 
features  in  common  with  these  systems. 

In  all  fairness,  the  MANPRINT  community  was  not  alone  in  failing  to  emphasize  the  potentially 
serious  impact  of  the  control  and  training  problems  on  Patriot  system  performance.  We  are  part 
of  a  team,  and  the  other  members  of  the  system  development  team — the  system’s  prime 
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contractor,  the  materiel  developer,  the  combat  developer,  the  TCM,  and  the  Army  test  and 
evaluation  (T&E)  community — all  overlooked  or  downplayed  these  issues.  These  observations 
regarding  Patriot  mostly  would  be  moot  were  it  not  for  the  fact  that  this  insensitivity  to  the 
downside  potential  associated  with  human  perfonnance  issues  continues  to  be  prevalent.  It  was 
observed  during  the  Patriot  Vigilance  project;  it  was  apparent  during  the  PDB-6  LUT  and  its 
aftennath;  and  it  continues  to  be  a  problem  in  AMD  systems  other  than  Patriot.  The  natural 
tendency  of  most  of  the  other  players  is  to  downplay  human  perfonnance  issues  as  isolated 
incidents  or  as  inconvenient,  troublesome  problems  with  no  clean  (i.e.,  technology-oriented) 
solution. 

In  the  case  of  Patriot  with  PDB-6,  a  strong  case  was  made  regarding  the  potential  seriousness  of 
the  control  and  training  issues.  Based  on  these  conclusions  and  supporting  material,  headway 
toward  developing  and  implementing  a  training  fix  also  was  made  in  the  Air  Defense  Artillery 
(ADA)  School  (see  Hawley,  Mares,  Fallin,  &  Wallet,  2007).  However,  the  request  for  a  PDB-6 
conditional  materiel  release  contained  the  usual  “all  is  well  with  the  system”  comments  and 
mostly  ignored  the  human  performance  problems  and  training  inadequacies  that  showed  up 
clearly  in  test  results  and  were  similar  to  those  that  were  judged  by  the  Anny  BOI  to  have 
contributed  to  the  Patriot  fratricides  during  OIF. 

Broader  Lessons  from  Patriot 

Beyond  the  specific  observations  stemming  from  our  review  of  Patriot  MANPRINT  materials 
going  back  some  20  years,  there  are  a  series  of  lessons  from  the  Patriot  experience  that 
generalize  to  present  and  future  system  development  efforts  other  than  Patriot.  Several  of  the 
more  prominent  of  these  broader  lessons  from  Patriot  are  now  briefly  discussed. 

Going-In  Concepts  Really  Matter.  The  first  general  lesson  to  be  taken  away  from  the  Patriot 
MANPRINT  experience  is  that  going-in  concepts  really  matter.  If  a  flawed  system  concept  is 
allowed  to  proceed  into  development,  it  may  prove  impossible  to  modify  that  concept  later  to 
obtain  necessary  levels  of  performance.  Faulty  going-in  concepts  pursued  into  development  can 
create  a  cul-de-sac  from  which  there  might  not  be  a  graceful  exit.  A  good  example  of  this 
situation  is  the  safety  veto  of  the  FAA’s  AAS  for  air  traffic  control  in  the  late  1990s  after  more 
than  $3.5  billion  had  been  spent  on  its  development.  Judging  from  available  documentation  on 
the  system’s  demise,  it  was  undone  by  a  combination  of  controller  reluctance  to  accept  the 
system  coupled  with  operational  test  results  suggesting  that  the  system  might  not  be  reliable 
enough  to  deploy  as  an  operational  air  traffic  control  system  (Bar- Yam,  1997). 

In  the  case  of  Patriot,  the  system’s  going-in  concept  flew  in  the  face  of  several  well-known  (even 
for  that  time)  dilemmas  concerning  automated  systems.  The  first  of  these  has  been  termed  the 
brittleness  problem  of  automata  (Woods,  2002).  The  brittleness  dilemma  refers  to  breakdowns 
in  handling  atypical  or  off-nominal  situations — similar  to  those  that  occurred  during  OIF. 
Patriot’s  developers  disregarded  the  brittleness  issue  and  proceeded  under  the  assumption  that 
automation  would  eliminate  many  of  the  human  performance  problems  associated  with  earlier 
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AMD  systems  and  also  reduce  training  requirements.  Both  assumptions  proved  to  be  unfounded. 
As  noted  previously,  automation  applied  to  real-time  C2  changes  the  operators’  roles  from 
traditional  operators  to  supervisory  controllers.  And  that  subtle  role  shift  brings  with  it  what  has 
been  tenned  the  Catch-22  of  human  supervisory  control  (Reason,  1990).  Consequently,  Patriot 
finds  itself  in  somewhat  the  same  situation  as  the  FAA’s  AAS.  The  system’s  developers  have 
committed  to  a  system  concept  that  demonstrates  patterns  of  perfonnance  unreliability  that  might 
prove  difficult  and  expensive  to  correct.  More  attention  to  proper  operator-system  function 
allocation  and  the  potential  downside  of  automation  as  the  system  evolved  might  have  resulted  in 
a  product  that  exhibited  fewer  of  these  problems. 

Training  Issues  Really  Matter.  The  second  general  lesson  emerging  from  the  Patriot 
MANPRINT  experience  is  that  training  issues  really  matter.  As  noted  previously,  while  the 
Patriot  system  evolved  over  its  25-year  life,  system  enhancements  (new  hardware  features,  new 
software  drops  like  PDB-6,  etc.)  the  system  became  more  complex.  Increased  complexity 
coupled  with  the  operator  role  shift  from  traditional  operator  to  supervisory  controller  added  to 
the  system’s  training  burden.  Yet  this  increase  in  training  requirements  was  not  matched  by 
corresponding  changes  in  institutional  or  unit  training.  Hawley,  Mares,  and  Giammanco  (2006) 
and  Hawley  and  Mares  (2007)  argue  that  changes  in  system  complexity  must  drive  quantitative 
and  qualitative  changes  in  the  associated  training  program.  Training  must  more  rigorous  and 
qualitatively  different  in  the  sense  that  it  emphasizes  the  development  of  mental  models  and 
knowledge  structures  and  not  simple  habit  transfer,  what  the  Army  BOI  referred  to  as  “rote 
drills.”  This  view  is  supported  by  other  observers  such  as  the  Defense  Science  Board.  In  the  first 
of  two  reports  on  training  for  future  conflicts,  the  DSB  (2001)  cautioned  that  an  increasing  risk 
exists  that  training  failures  will  negate  hardware  promise.  Their  2003  follow-on  report  further 
remarked  that  the  future  will  require  that  more  of  our  people  do  new  and  more  complicated 
things,  and  “meeting  this  challenge  amounts  to  a  qualitative  change  in  the  demands  placed  on  our 
people  that  cannot  be  supported  by  traditional  training  practices”  (DSB,  2003,  p.  38). 

These  observations  regarding  the  human  performance  impact  of  system  complexity  are  not 
particularly  new  or  unique  to  AMD.  For  example,  in  her  classic  work  In  the  Age  of  the  Smart 
Machine,  Shoshanna  Zuboff  (1988)  remarks  that  computer-mediated  work  like  that  found  in 
many  new  systems  brings  with  it  an  increase  in  “intellective  skill  requirements.”  Commenting 
on  what  they  had  observed  during  Operation  Desert  Storm  in  the  early  1990s,  Cordesman  and 
Wagner  (1996,  p.  25)  note  that  technical  advances  are  used  to  demand  more  from  operators,  and 
meeting  these  demands  often  requires  “exceptional  human  expertise.”  More  recently,  an  early 
operational  assessment  of  DARPA’s  Command  Post  of  the  Future  (CPOF)  currently  being  used 
by  the  Anny  in  Iraq  remarks  that  in  order  to  take  advantage  of  the  features  provided  by  this  new 
capability  there  is  a  “need  for  a  soldier  with  a  wider  ‘intellectual  bandwidth,’  where  management 
and  assimilation  of  information  from  many  sources  is  a  necessity”  (Center  for  Army  Lessons 
Learned,  2005,  p.  vii).  In  a  case  study  assessment  of  the  impact  of  net-centric  operations  using 
the  Stryker  Brigade  Combat  Team  (BCT)  as  an  exemplar,  Gonzales,  Johnson,  McEver,  Leedom, 
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Kingston,  and  Tseng  (2005,  p.  35)  concluded  that  “training  is  more  important  than  ever  in  the 
Stryker  brigade  and  other  digitized  units  because  the  networking  and  battle  command  systems 
employed  are  more  complex  than  those  used  in  analog-equipped  brigades.  If  soldiers  and 
commanders  are  not  adequately  trained  on  the  NCW  [network-centric  warfare]  systems  and  are 
not  proficient  in  their  use  in  stressful  battlefield  conditions,  then  these  NCW  systems  can  be  a 
hindrance  rather  than  a  help  in  combat.”  Finally,  in  a  post-test  briefing  to  the  Vice  Chief  of  Staff 
of  the  Army  concerning  the  Army  Battle  Command  and  Enablers  (ABCE)  system  of  systems,  the 
Army  Test  and  Evaluation  Command  (A TEC)  concluded  “There  is  no  indication  that  units  can 
dedicate  the  time,  resources,  or  personnel  to  adequately  train  on  the  digital  C4I  systems  and 
allow  the  unit  to  adequately  comprehend  the  system’s  capabilities,  much  less  exploit  these 
systems  as  a  force  multiplier”  (ATEC,  2006). 

So  what  does  all  of  this  have  to  do  with  MANPRINT?  The  answer  is  captured  in  the  two  DSB 
comments  cited  above.  First,  risk  exists  that  training  failures  will  negate  hardware  promise,  and 
second,  the  notion  that  new  systems  may  impose  a  training  burden  that  cannot  be  supported  by 
traditional  training  practices.  When  viewed  from  these  perspectives,  training  issues  are  indeed  a 
system  development  problem.  For  example,  in  the  case  of  Patriot  with  PDB-6,  the  Army 
Evaluation  Center’s  System  Assessment  Report  concluded  that  operator  perfonnance 
requirements  for  PDB-6  exceeded  the  current  Anny  training  standard.  This  conclusion  is 
significant  with  respect  to  downstream  system  perfonnance.  The  implication  is  that  if  training 
practices  remain  unchanged,  the  Patriot  system  with  PDB-6  has  become  too  complex  for  its 
operators  to  employ  effectively.  The  DSB’s  warning  that  training  failure  might  negate  hardware 
promise  will  have  become  a  reality. 

MANPRINT  analysts  and  assessors  must  forcefully  advocate  quantitative  and  qualitative  training 
changes  when  assessment  or  test  results  support  a  conclusion  like  that  put  forward  above  for 
ABCE  or  that  emerging  from  the  PDB-6  LUT.  The  operational  demonstration  of  new  training 
capabilities  and  methods  for  AMD  unit  training  reported  in  Hawley,  Mares,  Fallin,  and  Wallet 
(2007)  provides  an  example  of  the  kinds  of  positive  change  that  can  result  from  effective 
advocacy  by  the  MANPRINT  community.  Admittedly,  the  AMD  operational  demonstration  is  a 
tentative  first  step  in  the  direction  of  implementing  the  changes  that  will  be  necessary  to  bring 
AMD  unit  training  into  line  with  system  requirements,  but  it  is  a  first  step. 

Testing  Must  be  More  Comprehensive  and  Rigorous.  The  third  general  observation  emerging 
from  the  review  of  MANPRINT  practices  with  the  Patriot  system  is  that  operational  testing  must 
be  more  comprehensive  and  rigorous,  particularly  where  the  impact  of  human  performance  on 
system  performance  is  concerned.  During  the  PDB-6  LUT,  for  example,  the  MANPRINT 
support  team  encountered  a  strong  bias  on  the  part  of  testers  to  look  at  the  system  primarily  from 
a  hardware  and  software  perspective.  The  impact  of  operator  performance  on  system 
performance  was  regarded  as  a  secondary  issue — one  that  could  be  taken  care  of  mostly  with 
user-jury  type  questionnaires.  User-jury  type  questionnaires  are  potentially  useful  in  system 
evaluation,  but  they  should  not  be  the  sole  source  of  data  regarding  the  operators’  impact  on 
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system  performance.  Operators  who  are  not  familiar  with  new  system  functions  and 
requirements  often  “don’t  know  what  they  don’t  know,”  thus  their  subjective  assessments  may 
lack  critical  information  regarding  the  system’s  perfonnance  potential. 

Looking  back  on  the  situation  with  the  PDB-6  LUT  and  other  tests,  the  fact  that  MANPRINT 
(i.e.,  Soldier  perfonnance)  is  classified  as  a  supportability  issue  and  not  a  system  perfonnance 
issue  during  test  design  and  conduct  undoubtedly  contributes  to  the  hardware-software  bias  noted 
above.  Dekker  (2002)  argues  that  complex  systems  like  Patriot  and  others  coming  into  the  Army 
inventory  require  an  “overwhelming  human  contribution”  for  their  effective  operation.  He  goes 
on  to  state  that  “people  are  the  only  ones  who  can  hold  together  the  patchwork  of  technologies  in 
their  world;  the  only  ones  who  can  make  it  work  in  actual  practice”  (p.  103).  Given  this  view 
backed  up  with  the  observations  reported  in  the  previous  subsection,  it  is  hard  to  justify  the 
notion  that  Soldier  performance  is  a  secondary  issue  during  T&E.  Soldier  impact  on  system 
performance  cannot  continue  to  be  viewed  as  a  secondary  issue.  Continuing  to  do  so  opens  the 
door  to  total  system  reliability  problems  like  those  that  contributed  to  the  Patriot  fratricides 
during  OIF.  For  Patriot,  it  was  assumed  that  operators  would  be  able  to  compensate  for  known 
hardware-software  reliability  problems  in  the  area  of  track  classification  accuracy,  but  this 
assumption  was  never  verified  empirically  during  testing.  Later  events  during  OIF  indicated  that 
the  assumption  was  unwarranted. 

Meaningfully  putting  the  human  component  into  the  testing  equation  for  human-machine 
systems  will  require  a  significant  change  in  the  way  the  Army  prepares  test  players  to  participate 
in  operational  tests  (see  Hawley,  2007a).  In  present  usage,  the  term  “test  players”  refers  to  the 
individuals,  crews,  or  units  participating  in  a  test  event.  Both  the  Defense  Science  Board  and  the 
Government  Accountability  Office  (GAO)  have  criticized  the  Department  of  Defense  in  general, 
and  the  Army  in  particular,  for  not  properly  preparing  test  players  to  participate  in  operational 
tests  (e.g.,  DSB,  1999,  2003;  GAO,  2000).  The  DSB  has  stated  bluntly  that  the  “Army  continues 
to  field  new  equipment  without  adequate  training”  (DSB,  2003,  p.  44).  Training  deficiencies  for 
new  systems  are  a  problem  for  receiving  organizations,  but  they  can  be  remedied  with  use  over 
time.  However,  for  T&E,  pre-test  training  deficiencies  can  have  more  serious  consequences — 
because  T&E  frequently  is  a  one-shot  affair.  The  GAO  notes  that  testing  is  the  primary  means 
used  to  gauge  the  progress  being  made  when  a  concept  is  translated  into  an  actual  product  for 
users;  it  is  the  basis  for  determining  fitness  for  use  before  the  product  is  provided  to  users  (GAO, 
2000). 

Failures  in  the  training  domain  would  not  be  particularly  of  interest  to  testers  were  it  not  for  the 
serious  downstream  impact  of  inadequate  training  on  test  integrity.  The  GAO  and  DSB  both 
caution,  for  example,  that  inadequate  training  of  test  participants  seriously  undermines  the 
validity  of  test  results.  Following  a  review  of  a  representative  sample  of  operational  tests, 
Hawley  and  Frederickson  (1990)  concluded  that  inadequate  test  player  preparation  was  one  of 
the  most  frequent  reasons  for  test  failure.  Failure,  in  present  usage,  refers  to  an  inability  to 
cleanly  address  test  issues  using  test  data.  These  authors  cautioned  that  if  test  player  capabilities 
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are  uncertain,  test  results  are  likely  to  be  compromised.  The  most  common  form  of  compromise 
is  confounding  between  test  outcomes  and  pre-test  proficiency  levels.  It  is  not  possible  to  state 
unambiguously  that  test  outcomes  reflect  system  capabilities  and  features,  test  player 
proficiency,  or  some  combination  of  the  two. 

In  GAO’s  terminology,  poor  test  player  preparation  almost  inevitably  results  in  a  “hollow  test” 
(GAO,  2000,  p.  5).  A  hollow  test  is  one  that  satisfies  the  requirement  to  hold  a  testing  event,  but 
does  not  advance  system-related  knowledge.  Consequently,  test  results  provide  little  insight  into 
the  system’s  perfonnance  potential  and  cannot  be  used  to  debug  or  improve  the  system.  Test 
planners  often  attempt  to  compensate  for  an  emerging  hollow  test  by  restricting  or  scripting  test 
events  as  a  workaround  for  test  player  perfonnance  deficiencies.  A  great  deal  of  this  “re¬ 
scripting”  was  observed  during  the  PDB-6  LUT.  But  such  tactics  do  not  really  address  the  faulty 
train-up  issue.  Regardless  of  the  camouflage  used,  the  test  is  still  hollow.  Clever  statistics  and 
sophisticated  post- test  analyses  cannot  compensate  for  an  intrinsically  flawed  test.  For  test 
results  to  be  meaningful,  test  planners  must  verify  that  test  players  have  been  trained  to  the 
competency  levels  required  by  test  events. 

Lessons  Must  be  Learned.  Former  Chief  of  Staff  of  the  Army,  General  Eric  Shinseki,  is 
credited  with  the  observation  that  lessons  are  not  learned  until  a  resulting  change  occurs.  If 
change  does  not  occur,  the  lessons  are  merely  “observed.”  The  same  might  be  said  of 
MANPRINT  lessons  and  results.  In  the  case  of  Patriot  and  PDB-6,  the  MANPRINT  team 
encountered  enonnous  pressure  to  pass  the  system  on  and  support  a  conditional  materiel  release 
for  PDB-6  with  the  promise  that  problems  emerging  from  the  test  would  be  fixed  later.  This 
occurred  in  spite  of  the  fact  that  similar  problems  showing  up  in  earlier  tests  or  as  lessons  from 
combat  operations  during  OIF  had  not  yet  been  acted  upon.  Hard  and  unpleasant  facts  must  be 
faced  and  acted  upon  explicitly.  In  the  absence  of  action,  problems  typically  do  not  get  better 
with  time;  and  ignoring  problems  does  not  make  them  go  away. 

One  interesting  observation  emerging  from  the  PDB-6  test  concerns  the  handling  of  feedback 
from  operational  tests,  specifically  feedback  concerning  training,  doctrine,  or  procedural 
inadequacies  impacting  overall  system  performance.  Materiel  developers  (e.g.,  Program 
Managers)  routinely  get  feedback  from  T&E  and  are  accustomed  to  addressing  system 
inadequacies  based  on  test  results.  These  “wickets”  in  the  system  development  process  are 
routinely  used  and  well  developed.  Resource  or  other  constraints  may  prevent  the  timely 
implementation  of  essential  materiel  fixes,  but  the  mechanism  for  their  acknowledgement  is  well 
known. 

Such  is  not  the  case  for  training,  doctrine,  or  procedural  inadequacies — the  stuff  of  many 
MANPRINT  assessments.  In  the  case  of  training,  for  example,  feedback  regarding  training 
inadequacies  such  as  those  found  in  the  PDB-6  LUT  would  have  to  go  to  TRADOC  for 
institutional  training  fixes  and  to  various  Major  Commands  (e.g.,  Forces  Command — 
FORSCOM)  for  unit  training  modifications.  The  Army  Independent  Evaluator  judged  that  it  had 
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no  authority  to  task  TRADOC  or  FORSCOM  for  any  mitigation  plan  addressing  training  fixes. 
These  commands  could  be  apprised  of  potential  training  problems,  but  no  formal 
acknowledgement  of  the  problem  or  plan  for  its  remedy  is  required  in  response.  (Apparently, 
only  the  Anny  G3  has  the  authority  to  task  the  MACOMS.)  As  noted,  such  is  not  the  case  with 
respect  to  materiel  deficiencies.  PMs  are  routinely  required  to  provide  so-called  “get  well”  plans 
in  response  to  system  deficiencies  (per  AR  700-142).  The  get  well  plan  is  a  condition  for 
materiel  release,  and  the  progress  of  its  implementation  is  monitored  by  the  Independent 
Evaluator.  There  is  an  old  adage  that  what  gets  measured  gets  done.  Until  doctrine,  tactics, 
procedures,  training  methods,  training  support  (training  devices),  and  other  parts  of  the  total 
system  package  (e.g.,  the  DOTLM-PF  components,  minus  the  M)  are  put  on  the  same  plane  as 
materiel,  only  part  of  the  feedback  from  T&E  has  any  potential  for  long-term  beneficial  impact. 


MANPRINT  Going  Forward 


The  previous  section  noted  that  one  of  the  primary  motivators  for  this  report  was  the  TCM-LT’s 
musing  concerning  how  during  20  years  of  MANPRINT  work  on  Patriot  we  missed  many  of  the 
human  performance  and  training  problems  that  contributed  to  the  fratricides  during  OIF.  It 
cannot  be  stated  with  any  certainty  that  the  issues  of  effective  human  supervisory  control  and 
inadequate  training  were  completely  missed.  However,  it  is  certain  based  on  the  review  of 
available  MANPRINT  assessments  and  test  reports  that  a  discussion  of  these  issues  never  made 
it  into  those  documents.  Hence,  in  that  sense,  the  issues  were  missed.  In  retrospect,  one  must 
conclude  that  the  TCM’s  implied  criticism  of  the  Patriot  MANPRINT  program  was  justified. 

This  conclusion  raises  the  broader  question  of  why  these  issues  were  missed.  Part  of  the  answer 
lies  in  the  previous  observation  that  our  MANPRINT  methods  did  not  evolve  along  with  the 
system.  An  eroding  potential  for  effective  HSC  and  the  concomitant  problem  of  inadequate 
training  came  in  on  the  coattails,  so  to  speak,  of  technical  enhancements  to  the  Patriot  system 
over  time.  However,  we  kept  doing  what  we  had  always  done,  but  that  something  became 
increasingly  inadequate.  A  contributor  to  this  problem  lies  in  the  way  the  MANPRINT  program 
was  set  up  originally.  Back  in  the  1985-86  timeframe  when  the  program  was  codified  in  AR 
602-2,  the  breakout  of  the  MANPRINT  program  into  semi-independent  domains — HFE, 
Manpower,  Personnel,  Training,  Soldier  Survivability,  Health  Hazards,  and  Soldier 
Survivability — preserved  the  stakeholder  “rice  bowls”  that  existed  at  that  time.  However,  the 
all-important  issue  of  domain  integration  was  mostly  given  lip  service.  Platitudes  concerning  the 
importance  of  domain  integration  were  put  into  the  Regulation  and  supporting  documents,  but  no 
mechanism  was  put  in  place  to  bring  it  about.  The  lesson  from  Patriot  is  that  MANPRINT 
domain  integration  involves  more  than  a  simple  concatenation  of  assessments  from  the 
component  domains.  An  effective  MANPRINT  program  is  more  than  simple  the  sum  of  its 
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domain  parts.  In  Patriot,  operator  performance  issues  that  emerged  at  the  intersection  of  domains 
such  as  automation,  supervisory  control,  and  their  impact  on  operator  performance  demands  and 
training  requirements  simply  fell  through  the  cracks. 

A  second  observation  regarding  MANPRINT  going  forward  is  the  need  for  a  broader  analytical 
and  programmatic  perspective.  As  a  combined  work  program,  the  Patriot  Vigilance  project,  the 
PDB-6  LUT,  and  subsequent  efforts  at  implementation  such  as  the  training  operational 
demonstration,  support  and  illustrate  this  view.  Vicente  (2006)  and  others  advocate  what  might 
be  termed  a  sociotechnical  systems  (STS)  approach  to  MANPRINT  and  HSI.  An  STS  approach 
to  MANPRINT  begins  with  the  premise  that  people  (the  social  part  of  the  system)  and 
technology  must  be  viewed  as  an  integrated  system,  and  that  both  the  social  and  technical  parts 
of  the  system  must  be  optimized  equally  and  as  a  unit.  Nothing  of  this  sort  happened  with 
Patriot  as  it  evolved,  and  nothing  of  the  sort  is  happening  with  follow-on  AMD  systems.  In  spite 
of  General  Max  Thurman’s  admonitions  about  the  dangers  of  manning  systems  versus  equipping 
people  and  organizations  to  accomplish  a  mission,  we  still  mostly  build  systems  and  then  put 
people  into  them — and  then  put  those  systems  into  existing  organizations.  (Max  Thunnan  is 
generally  considered  to  be  the  “Godfather”  of  the  Army’s  MANPRINT  initiative.)  It  should  be 
noted  that  the  structure  of  the  Army’s  system  development  process  almost  guarantees  that  things 
will  work  this  way. 

Vicente’s  (2006)  notion  of  what  comprises  an  STS  approach  to  MANPRINT  and  system 
development  is  shown  in  figure  2.  All  of  the  steps  on  the  sociotechnical  ladder  are  necessary  for 
the  initiative  to  succeed  and  contribute  to  system  and  organizational  effectiveness.  Based  on  the 
Patriot  experience,  figure  2  also  suggests  that  issues  become  more  problematic  as  one  ascends 
the  ladder,  but  all  the  steps  are  necessary — all  the  way  to  the  top.  In  the  case  of  Patriot,  for 
example,  the  project  staff  is  discovering  that  issues  on  the  top  two  steps — Organization  and 
Political — seriously  impact  solutions  to  issues  at  lower  levels.  Implementing  the  training  fixes 
indicated  by  the  OIF  BOIs,  the  Patriot  Vigilance  project,  and  follow-on  work  is  impeded  by 
Patriot  and  AMD’s  organizational  culture  (the  way  we  routinely  do  things)  and  political  issues 
(e.g.,  TRADOC  and  MACOM  policies,  budgetary  pots,  regulations  at  various  levels,  stakeholder 
rice  bowls,  etc.).  As  MANPRINT  practitioners,  we  cannot  change  culture  or  affect  politics 
directly,  but  we  must  apprise  decision  makers  of  the  potential  impact  of  these  factors  on  system 
and  organizational  effectiveness. 
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Figure  2.  Ladder  of  sociotechnical  concerns. 
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